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ABSTRACT 

Two sets of multiple criterion diaensions were 
identified for use with disadvantaged adolescents who participated in 
youth-work training prograas* One set was coaposed of five factors 
applicable to short-term "success" at program coapletion (e.g., 
"Training Prograa Adjustaent"; "Social-Coaaunity Ad justaent") • six 
factors extracted fron a aatrix of post-prograa variables defined a 
criterion doaain covering various aspects of eaployaent and social 
adjustaent C*«g*» "General Adjustaent and Job Success"; "Blue-Collar 
Job Success"; "Coaaunity-Faaily Adjustaent"). Dependence on a single 
overriding criterion measure, such as eaployaent, was shown to livit 
the understanding of behavioral outcomes. (Author/KM) 
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T^-70 sets of multiple criterion dimensions T^rere identified for use vith 
disadvantaged adolescents who participated in youth-Vv^ork training programs. 
One set was composed of five factors appJdcable to short-term "success" at 
program completion (e.g^ "Ti-^aining Program Adjustment"; "Social-Coiivimnity 
Adjustment" )• Six factors extracted from a matrix of post-program variables 
defined a criterion domain covering various aspects of employment and social 
adjustment (e.g.^ "General Adjustment and Job Success"; "Blue-Collax Job 
Success"; "Community-Family Adjustment"). Limitations in understanding 
behavioral outcomes \/ere shovai^ where there is dependence on a single over- 
riding criterion measure such as Employraent, 
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Develojjment of Criterion Dimensions 
foz' Youth-VJoT'lc Training Pi'Ograiris 

Norman E. Freeberg 
Richard R. Re illy 

Educational Testing Service 
Princeton^ N. J. 

The process of criterion development has long been vievjed as ess'ential 
to the design and conduct of any social or educational prograan. Accompanying 
that vievr has been reasonable agreement among investigators regarding 
analytical approaches to the identification of multiple behavioral dimensions 
and their importance in offering better understanding of trainee performance 
and program effects (Dunnette; ,1963; Guion^ 19^57). Yet^ many national social, 
programs initiated during the past decade--such as those dealing vith manpovfer 
training --have neglected to develop viable criterion systems, v/i'thout which 
integrated research and evHluctl-loxt effoi-ts vvould seem to have been, scrioiwiy 
impaired (Borus & Tash^ 1970)* 

The present study represents an initial attempt to identify and .examine 
sets of empirically derived^, indei^endent criterion dim.ensions for a major 
class of manpovrer training programs designated as "youth -work training prograjns*' 
(Herman & Sadofsky^ I966). Such programs have served adolescents from poverty 
level backgrounds^ a large proportion of whom are also school dro^^outs and 
minority group members who have proven uniquely difficult to locate for data 
gathering purposes (u* S. Department of Labor ^ 1969)- 

Inherent in the above intent is an opportunity to describe ^salient 
performance characteristics of disadvantaged adolescents^ trained in these or 
similar programs^ from which more useful and meaningful definitions of training 
program ^'success" may be formulated. 



CrlV^'rion Oik Monnul ro:: : TVo crit«jrion qucstiojinaircs wore constructed. One 
dealt with rolrttivL?Xv : l'ov*;-tem (proximl) criterion variablcc applicable to 
trainee perfo Tm^co nt M rompl.-iion of the training prosrnm (Progran Com- 
pletion Quen* l«>u*:r*irc). other dealt with loncer term (diotai) outcomes 
applicable to tiic cx-trn'.!:o<.' come six months or more after coniplecion of the 
proi'^ram (Posi -rrocrani Qijr.^tionnairc). Bnsed on f.oals stated in enabling 
le^xislation ami intorvicvr wit)j progran; professionals regarding their training 
objectives, it in posriHl« to define five a priori (rational) categories that 
encompass broti'J areas of trainee r.uoces3 at program coraplctiou^ within which a 
total of ^2 criterion jnoMsures could be specified. These consist of: Voca- 
tional Plajinins Abilit.v (measures of level, appropriateness ajid quality of 
anticipated jobs and r^arcli motljods); Personal-Social Adjustment (suitability 
of relations vith fomllv, other conmunity members, police); Work Motivation 
(measures of v:illingnef;r. to accept employment in the face of adverse job 
features, wlllingnesf^ to take Job training); Vtork -Training Program Adjustment 
(gixidance counselor, v;ork supervisor and peer ratings, work site absences); 
and Peelings of Vocational Adeque-cy (trainee perception of his potential 
capabilities as tm employee). 

For the post-jMrogram criteria a total- of 36 variables was specified on 
the basis of the follovdng rationally defined categories: Extent and I/jvcl of 
Jinployment (hours worked i>er week, length of stoy on job, quality of Job); 
Job Perfonmincc and Stal^ility (number of jobs held, number and size of sale.ry 
increases, projno''uions, employer proficiency rating); Work Satisfaction (score 
on a job satisfaction scale); Personal-Social Adjustment (suitability of 
relations with family, police, other community members, healtli problems, 
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cavings, ur.c: of cioJlt.); and v;orJ:-roeking M-^^ivat.ion and ]'L'inninG (time 
required to find ri^i t. Job, munbcr of sources uj;od, nun^ber of Job iriU'i'vievs 
and applications filc^cl, maitabili'y of short- nnd long-term plans). 

Sample and bota Co]1 ciJon ; TJio Iro^rajn Cor.pletion Q^jcn^ionnoirc vas adninio- 
tore J to a r.aiTiX^le of Jiit;b vcl drupoji.r (% irAlcr-, fi? feirialca) ut the 
time they l»ad compl^^V^d their training in rix l^eighborhoocl Youth Corps, Out- 
of -School pi'ogra.'na IfKalcd in niy cities in various parts of the country. 
The Post-program QucijMonnaire v:ar, utilized with a sample of 15** former 
trainees {(O mIcs, females), fron the same training programs in the sajac 
six cities, \:ho had bocn out of the program for periods ranging from six 
months t* 1 l/2 yearn. All member*? of the latter sample had obtained full- 
time employment for at least one \;cek or move after leaving the program. 

Mean respondent a/',u vas l6.7 years for the program completion sample and 
Ifl.l years for the poy t-program .sunple. Ethnic composition of the samples^ 
\ms approximately 8o;f l^ack, Of^ Spanish-speaking (iMerto Rican or Mexican- 
American) and Ik^ V/hitc. 

The quest ionnaii^cs were orally' administered to these cross-sectional 
samples on an individual basis by si;: interviewers, each of whom were Black 
and residing in the city in wliich they collected the data. Respondents 
were paid for their participation • 

RESULTS 

Each of the two nc^ts of variables from each of the t\ro criterion question- 
naires (^9 Pi'ogram Coirplotion and % post-ft*ogram variables) were intercorrelatcd 
and factor analyzed using a principal components solution vrith varimox rotation 
to orthogonality* 
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Program. C!o:nT33.etion Criterion DiixTifiionn - as derived from the 'jP variable.?, 
consisted of five interprctable factors fairly similar to the hypothesized, 
a priori I catefjorien. TJic dominatit factor was clearly identifiable as a 
Traininr, Trocram Ad/^'stnent factor in which the pattern of londincfi deplete d 
the trnlnop v?ho received hirh^v nroficirnry ratingr, fron coxjnnrlorr ruid \ror!% 
cupervisorn, hail fev:er work-citf absences, v;as willinc to accept training in 
a greater variety of Jobs and tended not to liave problems with the police (self 
report ajid actual police records). A second factor defined a cluster of 
behaviors that is primarily external to performance in the training proRran nnd 
was dcsifinated as Social-Co?nnunity Ad^justment wit)i outcomes of better famil^v 
relations, fewer problems \d.tli people in the comunity, less trouble with police, 
and a higher peer ratine score. 

IVo other factors were based largely on trainee personal perceptions. One 
way denititiHled as Jub Ayplj-uMmi Ltfvel wlLh Uie pu:»iLive cnJ uf the faotoi" 
defining Greater expectations for starting and higliost salaries, a better qual- 
ity first job desired, more suitable long range plans, higher self appraisal, 
of ability to iccomplish plans and higher scores on feelings of vocational 
adequacy and a VJork Motivation Scale. The other factor was identified as Job 
Planniaq Competency since it combined more positive feelings of ability to do 
a sought-after job, with greater knowledge of the job sought and more and better 
ways to undertake a job search. 

The last of the program completion factors was one of Interpersonal 
Adjustment with lead loadings indicating more frequent selection for rating 
by peers and higher peer rating scores. In addition there are positive loadings 
on feelings of vocational adequacy, concern with Job characteristics and health 
status. 
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poaf «>lrocra!n CriV^'lon Dlr^cn^^Jonr - fi'cxn €he jt y nntrix of jor t-pro^^.rnjn 
iaeasv2*c*c, differed rr^rlcedly from 1-hc Jiyj/otliosisod dij7.ejicjcnfl. 'i'ijo dondnant 
factor of the six extracted v;ns flcfincfl ac (;enera1 A<^K]u5tnieht r',nd Job Snccecn 
with the key varlahloa in the loadinc pnt<.crn depicting the rxiccescful ex- 
trainoc one \:hv vac cnpJo.vcfl (on t>r c]a.v of t]:< iniicrvie;;), r-coivcd a 
higher employer ratine, had ntaycd on Ids ,iob lonrer, more nearly met his job 
expectations, got along bett'-r \rith hi:: ftuaiiy, saved money, ajid had fcv;er 
problems vdlth police. Interestingly, this factor tended to depict outcomes 
applicable to ex-trainees v:ho took white collar jobs. However, a separate 
factor of Blue Collar Job Success also, emerged, vrith those wlio entered blue 
collar employment as the onen vho found their first job more quiclQy, worked 
more hours per v:cek, received a liigher starting salary, obtained a liigher level 
job and expressed greater job satisfaction. 

A third job-related factor was defined as Job fitability 'Mobility v:itli more 
mobile former trainees (in jobs and residences) tending to be i*ated lo;:cr in 
proficiency by '^.le employer. 

Of the three remaining factors, one clearly defined Comm\inity and Family 
Adjustment with the ex-trainee having fev^er problents with the police, providing 
more financial assistance to the family, having better personal relations v;ith 
his family' and with other members of tlie community, and being more readily able 
to obtain credit. The designation of a fifth factor us Strivinn for Vocational 
Success could be inferred from favorable outcomes in a number of on-the-job 
criteria (higher raises, more promotions, higher employer ratings) and in the 
ex-trainee's desire for higher levels of employmonl. The remaining factor 
designated as Vocational Adjustment-Dissatisfaction encompassed outcomes that 
describe aspects of vocational adjustment (higher employer rating, good short 
term job planning, more raises) but tended, despite the successes, to be 
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nccompanlei by cx-trainee fcc3in/^n of job dinafitjsfnctioti (i^v*, hv tend«vl 

not to have mot his job expectations and to score lo\;er on the Jc b Satis fu^'tion 

Scale)* 

In the above poat-prograjn analysis, the Jinportancc of Inr- c''i.*«'M*atin|y jcb* 
related critc-rion variables neccr.sits^tod n^se of a co?rplt: with crployment 
experience • But a question can be posed of the value of the dicliotooouc 
employment criterion per se (i*c«, Obtained Post-Pfograa Employment vi« Hever 
Employed) in terras of its pattern of relationships to other critcr3on ovlccacs* 
A sample of 88 former trainees v;ho had never obtained employment \:cre pen: I'.- ted 
to respond to those portions of the questionnaire pertinent to their post- 
program experiences • These variables were, of course, limited solely to thv 
planninc-niotivational-adjustmcnt types of outcomes • Correlations vere co-T-pulci 
between the dichotomous employment measure and 1^* suc)i applicable questionnaire 

out« No meaningful pattern of relationships between tJie ih variables and the 
dichotomous employment outcome was founds I'hc only correlation of note v/a:J a 
partial r of .1^0 (p < •Ol) between Employment and Quality of Job Soxight (i.e., 
those former trainees who obtained full-time employment tended to seek hichov 
quality jobs than those who had never obtained post-program employment). 

CONCLUSIOIiS 

Coherent and sensibly interpretable criterion dimensions could be derived 
empirically for use with youth-work training program enrollees at compleidon of 
the training program and over longer post-program periods, Tlie structure of 
the criterion domain identified points up the multiple nature of outcomes 
needed to define complex behavioral aspects of trainee "succesn." By contrast, 
a single, overriding criterion (i«e#. Employment vs. Mo EmploiTiient ), whatever its 
face value, is highly limiting in any contribution to such understanding. 
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Uic proxjjpal and distal diiwnnloiii: er.trr.iicA in cross-rcii.ional 
Btu«ly provide a flexible core of criterion varlnbleo (r.r ractorcd scales) 
that miglit onliance much needed continuity of research efforts with disadvan^ 
tagcd, minority youngsters enrolled in manpo-.vi-r traininn programs. 



Refercnccn 



Rorus, M. & Tosh, VJt Measurinc the impact of inanpcn;or prosrojiw: A primer* 

Ann Arbor, Michigan: Inntitutc of Labor and Industrial Uolationn, 19T0. 
Dunncttc, M. A note on the criterion* Joiimal of Ai^plied P^ycholo/yj 19^^5, 

Guion, R. personnel selection. Annual Review of Psycliolony , 196?, l6, 191-P16. 
Herman, M, & Sadofsky, S. Youth'^vork programs . New York: Center for the Study 

of Unemployed Youth, Graduate School of Social VJork, Nev York University, 

1966. 

United States Department of I^bor. A compilation of papers on data gatherinc 
in manpower research, Office of ltonpov:er Research, Washington, D. C. , 
September 1969. (Unpublished manuscript) 



